Detection of latency, packet drops, and network hops through a tunnel by tracing hops therein

ABSTRACT

Techniques for using traceroute with tunnels and cloud-based systems for determining measures of network performance are presented. Systems and methods include receiving a request, from a client, for a trace of the tunnel; causing the trace inside the tunnel; obtaining results of the trace inside the tunnel; and sending the results of the trace inside the tunnel to the client so that the client aggregates these details with details from one or more additional legs to provide an overall view of a service path between the client and a destination.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to networking and computing.More particularly, the present disclosure relates to various techniquesfor using traceroute with tunnels and cloud-based systems fordetermining measures of network performance.

BACKGROUND OF THE DISCLOSURE

The trend in Information Technology (IT) includes applications andresources being located in the cloud, users working from home oranywhere, and for the Internet becoming the new corporate network. Assuch, there is a need for IT to monitor and isolate issues affecting theuser-to-cloud application experience. That is, there is a need for ITadministrators to have visibility into the network and details ofservice paths. Various User Experience (UX or UEX) tools and approachesseek to quantify an individual user's satisfaction with a product orservice such as a networking application. Examples include End UserExperience Monitoring (EUEM) tools, Network Performance Monitoring andDiagnostics Market (NPMD) tools, Application Performance Monitoring(APM) tools, and Digital Experience Monitoring (DEM) tools.

One aspect of these tools includes network measurements, such as using atraceroute. There are various tools to collect information aboutpossible network paths followed by traffic. These tools give a holisticview of the complete network path. Traceroute is used “trace routes” ofpaths and measure packet delays in Internet Protocol (IP) networks.Traceroute is a diagnostic command to find the routes (paths) andmeasures the latency to each hop. In traceroute, each node is called ahop, and the latency is the round trip from the user's machine to thehop.

With increasing network sniffing and network threats, many enterprisesuse encrypted tunnels. Also, cloud-based systems can forward traffic tothe cloud using encrypted tunnels. For example, tunnels can be based onGeneric Routing Encapsulation (GRE), Layer Two Tunneling Protocol(L2TP), Internet Protocol (IP) Security (IPsec), Virtual PrivateNetworks (VPN), Hypertext Transfer Protocol (HTTP), Transmission ControlProtocol (TCP), etc. Some of these tunnels are opaque to existing toolssuch as traceroute, causing the wrong details about network state.

Also, conventional TCP traceroute applications/tools cannot determine ifthe destination has been reached as they have no ability to read theresponse sent by the destination. The conventional traceroute haslimitations that it might not be complete, and the results are notaccurate for the final hop as the final hop does not provide theprocessing delay. The traceroute results might not be complete as thefinal destination might not respond to the probe. The conventionaltraceroute does not provide the latency between the hops. Routerstypically have a very fast forward path as this is done in the hardware,but some routers take significant time to respond to Time to Live (TTL)expired messages as they do this through software.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure relates to various techniques for usingtraceroute with tunnels and cloud-based systems for determining measuresof network performance. The various techniques are used to detectnetwork hops, packet loss, and latency from a client to a destination aswell as discover how the client connects to the Internet and if anyproxies or firewalls are present in the path. For determining aconnection to the Internet, the present disclosure includes a techniqueto detect tunnels. For determining proxies or firewalls, the presentdisclosure utilizes an Application Programming Interface (API) to detectan egress router's IP port on a client's network. Once the client hasvisibility of the path (i.e., tunnels, proxies, firewalls, etc.), theclient can communicate, such as out of band, to request other devices totrace different legs. Note, in various descriptions, the term tracerouteor trace can also include PING, such as the My Traceroute (MTR).

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIG. 1 is a network diagram of a cloud-based system offering security asa service;

FIG. 2 is a network diagram of an example implementation of thecloud-based system;

FIG. 3 is a block diagram of a server that may be used in thecloud-based system of FIGS. 1 and 2 or the like;

FIG. 4 is a block diagram of a user device that may be used with thecloud-based system of FIGS. 1 and 2 or the like;

FIG. 5 is a network diagram of the cloud-based system illustrating anapplication on user devices with users configured to operate through thecloud-based system;

FIG. 6 is a network diagram of a Zero Trust Network Access (ZTNA)application utilizing the cloud-based system of FIGS. 1 and 2;

FIG. 7 is a network diagram of the cloud-based system of FIGS. 1 and 2in an application of digital experience monitoring;

FIG. 8 is a network diagram of the cloud-based system of FIGS. 1 and 2with various cloud tunnels, labeled as cloud tunnels, for forwardingtraffic;

FIGS. 9 and 10 are flow diagrams of a cloud tunnel illustrating acontrol channel (FIG. 9) and a data channel (FIG. 10), with the tunnelillustrated between a client and a server;

FIG. 11 is a network diagram of a traceroute between a user and adestination with no tunnel in between;

FIG. 12 is a network diagram of a traceroute between a user and adestination with an opaque tunnel between a tunnel client and a tunnelserver;

FIG. 13 is a flowchart of a process for detecting a tunnel between auser device and a destination;

FIG. 14 is a flowchart of a process for collecting network details in atrace where there is an opaque tunnel;

FIG. 15 is a flow diagram illustrating actions between the client (userdevice), the tunnel client, the egress router, the tunnel server, andthe destination (node) in an example operation of the process of FIGS.13 and 14;

FIG. 16 is a flowchart of a process for detection of network hops andlatency through an opaque tunnel and detection misconfiguration oftunnels;

FIG. 17 is a flowchart of a process for detection of latency, packetdrops, and network hops through a TCP tunnel using ICMP and UDP probes;

FIG. 18 is a flowchart of a process for detection of latency, packetdrops, and network hops through a tunnel by tracing hops therein;

FIG. 19 is a network diagram illustrating a user connected to anenforcement node in a digital experience monitoring application;

FIG. 20 is a flow diagram illustrating actions between the client (userdevice), the application, the egress router, the enforcement node, andthe destination in an example operation of the process of FIGS. 13 and14, along with caching of trace results at the enforcement node;

FIG. 21 is a flowchart of a process for metric computation fortraceroute probes using cached data to prevent a surge on destinationservers;

FIG. 22 is a flowchart of a process for TCP traceroute using RST andSYN-ACK to determine destination reachability;

FIG. 23 is a network diagram with an excerpt of the network diagram ofFIG. 19 illustrating Legs 2 and 3 for illustrating adaptive probing;

FIG. 24 is a flowchart of an adaptive probe process for tracerouteprobes;

FIG. 25 is a network diagram of a network for illustrating an averagelatency calculation;

FIG. 26 is a diagram of the network of FIG. 25 illustrating anoperation;

FIGS. 27-30 illustrate an example operation of the average latencyadjustment.

FIGS. 31-34 illustrate an example operation of the differential averagelatency adjustment; and

FIG. 35 is a flowchart of a process for an accurate differentialtraceroute latency calculation between hops.

DETAILED DESCRIPTION OF THE DISCLOSURE

Again, the present disclosure relates to various techniques for usingtraceroute with tunnels and cloud-based systems for determining measuresof network performance. The various techniques are used to detectnetwork hops, packet loss, and latency from a client to a destination aswell as discover how the client connects to the Internet and if anyproxies or firewalls are present in the path. For determining aconnection to the Internet, the present disclosure includes a techniqueto detect tunnels. For determining proxies or firewalls, the presentdisclosure utilizes an Application Programming Interface (API) to detectan egress router's IP port on a client's network. Once the client hasvisibility of the path (i.e., tunnels, proxies, firewalls, etc.), theclient can communicate, such as out of band, to request other devices totrace different legs. Note, in various descriptions, the term tracerouteor trace can also include PING, such as the My Traceroute (MTR).

§ 1.0 Example Cloud-Based System Architecture

FIG. 1 is a network diagram of a cloud-based system 100 offeringsecurity as a service. Specifically, the cloud-based system 100 canoffer a Secure Internet and Web Gateway as a service to various users102, as well as other cloud services. In this manner, the cloud-basedsystem 100 is located between the users 102 and the Internet as well asany cloud services 106 (or applications) accessed by the users 102. Assuch, the cloud-based system 100 provides inline monitoring inspectingtraffic between the users 102, the Internet 104, and the cloud services106, including Secure Sockets Layer (SSL) traffic. The cloud-basedsystem 100 can offer access control, threat prevention, data protection,etc. The access control can include a cloud-based firewall, cloud-basedintrusion detection, Uniform Resource Locator (URL) filtering, bandwidthcontrol, Domain Name System (DNS) filtering, etc. The threat preventioncan include cloud-based intrusion prevention, protection againstadvanced threats (malware, spam, Cross-Site Scripting (XSS), phishing,etc.), cloud-based sandbox, antivirus, DNS security, etc. The dataprotection can include Data Loss Prevention (DLP), cloud applicationsecurity such as via a Cloud Access Security Broker (CASB), file typecontrol, etc.

The cloud-based firewall can provide Deep Packet Inspection (DPI) andaccess controls across various ports and protocols as well as beingapplication and user aware. The URL filtering can block, allow, or limitwebsite access based on policy for a user, group of users, or entireorganization, including specific destinations or categories of URLs(e.g., gambling, social media, etc.). The bandwidth control can enforcebandwidth policies and prioritize critical applications such as relativeto recreational traffic. DNS filtering can control and block DNSrequests against known and malicious destinations.

The cloud-based intrusion prevention and advanced threat protection candeliver full threat protection against malicious content such as browserexploits, scripts, identified botnets and malware callbacks, etc. Thecloud-based sandbox can block zero-day exploits (just identified) byanalyzing unknown files for malicious behavior. Advantageously, thecloud-based system 100 is multi-tenant and can service a large volume ofthe users 102. As such, newly discovered threats can be promulgatedthroughout the cloud-based system 100 for all tenants practicallyinstantaneously. The antivirus protection can include antivirus,antispyware, antimalware, etc. protection for the users 102, usingsignatures sourced and constantly updated. The DNS security can identifyand route command-and-control connections to threat detection enginesfor full content inspection.

The DLP can use standard and/or custom dictionaries to continuouslymonitor the users 102, including compressed and/or SSL-encryptedtraffic. Again, being in a cloud implementation, the cloud-based system100 can scale this monitoring with near-zero latency on the users 102.The cloud application security can include CASB functionality todiscover and control user access to known and unknown cloud services106. The file type controls enable true file type control by the user,location, destination, etc. to determine which files are allowed or not.

For illustration purposes, the users 102 of the cloud-based system 100can include a mobile device 110, a headquarters (HQ) 112 which caninclude or connect to a data center (DC) 114, Internet of Things (IoT)devices 116, a branch office/remote location 118, etc., and eachincludes one or more user devices (an example user device 300 isillustrated in FIG. 5). The devices 110, 116, and the locations 112,114, 118 are shown for illustrative purposes, and those skilled in theart will recognize there are various access scenarios and other users102 for the cloud-based system 100, all of which are contemplatedherein. The users 102 can be associated with a tenant, which may includean enterprise, a corporation, an organization, etc. That is, a tenant isa group of users who share a common access with specific privileges tothe cloud-based system 100, a cloud service, etc. In an embodiment, theheadquarters 112 can include an enterprise's network with resources inthe data center 114. The mobile device 110 can be a so-called roadwarrior, i.e., users that are off-site, on-the-road, etc. Those skilledin the art will recognize a user 102 has to use a corresponding userdevice 300 for accessing the cloud-based system 100 and the like, andthe description herein may use the user 102 and/or the user device 300interchangeably.

Further, the cloud-based system 100 can be multi-tenant, with eachtenant having its own users 102 and configuration, policy, rules, etc.One advantage of the multi-tenancy and a large volume of users is thezero-day/zero-hour protection in that a new vulnerability can bedetected and then instantly remediated across the entire cloud-basedsystem 100. The same applies to policy, rule, configuration, etc.changes—they are instantly remediated across the entire cloud-basedsystem 100. As well, new features in the cloud-based system 100 can alsobe rolled up simultaneously across the user base, as opposed toselective and time-consuming upgrades on every device at the locations112, 114, 118, and the devices 110, 116.

Logically, the cloud-based system 100 can be viewed as an overlaynetwork between users (at the locations 112, 114, 118, and the devices110, 116) and the Internet 104 and the cloud services 106. Previously,the IT deployment model included enterprise resources and applicationsstored within the data center 114 (i.e., physical devices) behind afirewall (perimeter), accessible by employees, partners, contractors,etc. on-site or remote via Virtual Private Networks (VPNs), etc. Thecloud-based system 100 is replacing the conventional deployment model.The cloud-based system 100 can be used to implement these services inthe cloud without requiring the physical devices and management thereofby enterprise IT administrators. As an ever-present overlay network, thecloud-based system 100 can provide the same functions as the physicaldevices and/or appliances regardless of geography or location of theusers 102, as well as independent of platform, operating system, networkaccess technique, network access provider, etc.

There are various techniques to forward traffic between the users 102 atthe locations 112, 114, 118, and via the devices 110, 116, and thecloud-based system 100. Typically, the locations 112, 114, 118 can usetunneling where all traffic is forward through the cloud-based system100. For example, various tunneling protocols are contemplated, such asGRE, L2TP, IPsec, customized tunneling protocols, etc. The devices 110,116, when not at one of the locations 112, 114, 118 can use a localapplication that forwards traffic, a proxy such as via a ProxyAuto-Config (PAC) file, and the like. An application of the localapplication is the application 350 described in detail herein as aconnector application. A key aspect of the cloud-based system 100 is alltraffic between the users 102 and the Internet 104 or the cloud services106 is via the cloud-based system 100. As such, the cloud-based system100 has visibility to enable various functions, all of which areperformed off the user device in the cloud.

The cloud-based system 100 can also include a management system 120 fortenant access to provide global policy and configuration as well asreal-time analytics. This enables IT administrators to have a unifiedview of user activity, threat intelligence, application usage, etc. Forexample, IT administrators can drill-down to a per-user level tounderstand events and correlate threats, to identify compromiseddevices, to have application visibility, and the like. The cloud-basedsystem 100 can further include connectivity to an Identity Provider(IDP) 122 for authentication of the users 102 and to a SecurityInformation and Event Management (SIEM) system 124 for event logging.The system 124 can provide alert and activity logs on a per-user 102basis.

FIG. 2 is a network diagram of an example implementation of thecloud-based system 100. In an embodiment, the cloud-based system 100includes a plurality of enforcement nodes (EN) 150, labeled asenforcement nodes 150-1, 150-2, 150-N, interconnected to one another andinterconnected to a central authority (CA) 152. The nodes 150 and thecentral authority 152, while described as nodes, can include one or moreservers, including physical servers, virtual machines (VM) executed onphysical hardware, etc. An example of a server is illustrated in FIG. 4.The cloud-based system 100 further includes a log router 154 thatconnects to a storage cluster 156 for supporting log maintenance fromthe enforcement nodes 150. The central authority 152 provide centralizedpolicy, real-time threat updates, etc. and coordinates the distributionof this data between the enforcement nodes 150. The enforcement nodes150 provide an onramp to the users 102 and are configured to executepolicy, based on the central authority 152, for each user 102. Theenforcement nodes 150 can be geographically distributed, and the policyfor each user 102 follows that user 102 as he or she connects to thenearest (or other criteria) enforcement node 150. Of note, thecloud-based system is an external system meaning it is separate fromtenant's private networks (enterprise networks) as well as from networksassociated with the devices 110, 116, and locations 112, 118.

The enforcement nodes 150 are full-featured secure internet gatewaysthat provide integrated internet security. They inspect all web trafficbi-directionally for malware and enforce security, compliance, andfirewall policies, as described herein, as well as various additionalfunctionality. In an embodiment, each enforcement node 150 has two mainmodules for inspecting traffic and applying policies: a web module and afirewall module. The enforcement nodes 150 are deployed around the worldand can handle hundreds of thousands of concurrent users with millionsof concurrent sessions. Because of this, regardless of where the users102 are, they can access the Internet 104 from any device, and theenforcement nodes 150 protect the traffic and apply corporate policies.The enforcement nodes 150 can implement various inspection enginestherein, and optionally, send sandboxing to another system. Theenforcement nodes 150 include significant fault tolerance capabilities,such as deployment in active-active mode to ensure availability andredundancy as well as continuous monitoring.

In an embodiment, customer traffic is not passed to any other componentwithin the cloud-based system 100, and the enforcement nodes 150 can beconfigured never to store any data to disk. Packet data is held inmemory for inspection and then, based on policy, is either forwarded ordropped. Log data generated for every transaction is compressed,tokenized, and exported over secure Transport Layer Security (TLS)connections to the log routers 154 that direct the logs to the storagecluster 156, hosted in the appropriate geographical region, for eachorganization. In an embodiment, all data destined for or received fromthe Internet is processed through one of the enforcement nodes 150. Inanother embodiment, specific data specified by each tenant, e.g., onlyemail, only executable files, etc., is processed through one of theenforcement nodes 150.

Each of the enforcement nodes 150 may generate a decision vector D=[d1,d2, . . . , dn] for a content item of one or more parts C=[c1, c2, . . ., cm]. Each decision vector may identify a threat classification, e.g.,clean, spyware, malware, undesirable content, innocuous, spam email,unknown, etc. For example, the output of each element of the decisionvector D may be based on the output of one or more data inspectionengines. In an embodiment, the threat classification may be reduced to asubset of categories, e.g., violating, non-violating, neutral, unknown.Based on the subset classification, the enforcement node 150 may allowthe distribution of the content item, preclude distribution of thecontent item, allow distribution of the content item after a cleaningprocess, or perform threat detection on the content item. In anembodiment, the actions taken by one of the enforcement nodes 150 may bedeterminative on the threat classification of the content item and on asecurity policy of the tenant to which the content item is being sentfrom or from which the content item is being requested by. A contentitem is violating if, for any part C=[c1, c2, . . . , cm] of the contentitem, at any of the enforcement nodes 150, any one of the datainspection engines generates an output that results in a classificationof “violating.”

The central authority 152 hosts all customer (tenant) policy andconfiguration settings. It monitors the cloud and provides a centrallocation for software and database updates and threat intelligence.Given the multi-tenant architecture, the central authority 152 isredundant and backed up in multiple different data centers. Theenforcement nodes 150 establish persistent connections to the centralauthority 152 to download all policy configurations. When a new userconnects to an enforcement node 150, a policy request is sent to thecentral authority 152 through this connection. The central authority 152then calculates the policies that apply to that user 102 and sends thepolicy to the enforcement node 150 as a highly compressed bitmap.

The policy can be tenant-specific and can include access privileges forusers, websites and/or content that is disallowed, restricted domains,DLP dictionaries, etc. Once downloaded, a tenant's policy is cacheduntil a policy change is made in the management system 120. The policycan be tenant-specific and can include access privileges for users,websites and/or content that is disallowed, restricted domains, DLPdictionaries, etc. When this happens, all of the cached policies arepurged, and the enforcement nodes 150 request the new policy when theuser 102 next makes a request. In an embodiment, the enforcement node150 exchange “heartbeats” periodically, so all enforcement nodes 150 areinformed when there is a policy change. Any enforcement node 150 canthen pull the change in policy when it sees a new request.

The cloud-based system 100 can be a private cloud, a public cloud, acombination of a private cloud and a public cloud (hybrid cloud), or thelike. Cloud computing systems and methods abstract away physicalservers, storage, networking, etc., and instead offer these as on-demandand elastic resources. The National Institute of Standards andTechnology (NIST) provides a concise and specific definition whichstates cloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned and released with minimal management effort orservice provider interaction. Cloud computing differs from the classicclient-server model by providing applications from a server that areexecuted and managed by a client's web browser or the like, with noinstalled client version of an application required. Centralizationgives cloud service providers complete control over the versions of thebrowser-based and other applications provided to clients, which removesthe need for version upgrades or license management on individual clientcomputing devices. The phrase “Software as a Service” (SaaS) issometimes used to describe application programs offered through cloudcomputing. A common shorthand for a provided cloud computing service (oreven an aggregation of all existing cloud services) is “the cloud.” Thecloud-based system 100 is illustrated herein as an example embodiment ofa cloud-based system, and other implementations are also contemplated.

As described herein, the terms cloud services and cloud applications maybe used interchangeably. The cloud service 106 is any service madeavailable to users on-demand via the Internet, as opposed to beingprovided from a company's on-premises servers. A cloud application, orcloud app, is a software program where cloud-based and local componentswork together. The cloud-based system 100 can be utilized to provideexample cloud services, including Zscaler Internet Access (ZIA), ZscalerPrivate Access (ZPA), and Zscaler Digital Experience (ZDX), all fromZscaler, Inc. (the assignee and applicant of the present application).Also, there can be multiple different cloud-based systems 100, includingones with different architectures and multiple cloud services. The ZIAservice can provide the access control, threat prevention, and dataprotection described above with reference to the cloud-based system 100.ZPA can include access control, microservice segmentation, etc. The ZDXservice can provide monitoring of user experience, e.g., Quality ofExperience (QoE), Quality of Service (QoS), etc., in a manner that cangain insights based on continuous, inline monitoring. For example, theZIA service can provide a user with Internet Access, and the ZPA servicecan provide a user with access to enterprise resources instead oftraditional Virtual Private Networks (VPNs), namely ZPA provides ZeroTrust Network Access (ZTNA). Those of ordinary skill in the art willrecognize various other types of cloud services 106 are alsocontemplated. Also, other types of cloud architectures are alsocontemplated, with the cloud-based system 100 presented for illustrationpurposes.

§ 2.0 User Device Application for Traffic Forwarding and Monitoring

FIG. 3 is a network diagram of the cloud-based system 100 illustratingan application 350 on user devices 300 with users 102 configured tooperate through the cloud-based system 100. Different types of userdevices 300 are proliferating, including Bring Your Own Device (BYOD) aswell as IT-managed devices. The conventional approach for a user device300 to operate with the cloud-based system 100 as well as for accessingenterprise resources includes complex policies, VPNs, poor userexperience, etc. The application 350 can automatically forward usertraffic with the cloud-based system 100 as well as ensuring thatsecurity and access policies are enforced, regardless of device,location, operating system, or application. The application 350automatically determines if a user 102 is looking to access the openInternet 104, a SaaS app, or an internal app running in public, private,or the datacenter and routes mobile traffic through the cloud-basedsystem 100. The application 350 can support various cloud services,including ZIA, ZPA, ZDX, etc., allowing the best in class security withzero trust access to internal apps. As described herein, the application350 can also be referred to as a connector application.

The application 350 is configured to auto-route traffic for seamlessuser experience. This can be protocol as well as application-specific,and the application 350 can route traffic with a nearest or best fitenforcement node 150. Further, the application 350 can detect trustednetworks, allowed applications, etc. and support secure network access.The application 350 can also support the enrollment of the user device300 prior to accessing applications. The application 350 can uniquelydetect the users 102 based on fingerprinting the user device 300, usingcriteria like device model, platform, operating system, etc. Theapplication 350 can support Mobile Device Management (MDM) functions,allowing IT personnel to deploy and manage the user devices 300seamlessly. This can also include the automatic installation of clientand SSL certificates during enrollment. Finally, the application 350provides visibility into device and app usage of the user 102 of theuser device 300.

The application 350 supports a secure, lightweight tunnel between theuser device 300 and the cloud-based system 100. For example, thelightweight tunnel can be HTTP-based. With the application 350, there isno requirement for PAC files, an IPsec VPN, authentication cookies, oruser 102 setup.

§ 3.0 Example Server Architecture

FIG. 4 is a block diagram of a server 200, which may be used in thecloud-based system 100, in other systems, or standalone. For example,the enforcement nodes 150 and the central authority 152 may be formed asone or more of the servers 200. The server 200 may be a digital computerthat, in terms of hardware architecture, generally includes a processor202, input/output (I/O) interfaces 204, a network interface 206, a datastore 208, and memory 210. It should be appreciated by those of ordinaryskill in the art that FIG. 4 depicts the server 200 in an oversimplifiedmanner, and a practical embodiment may include additional components andsuitably configured processing logic to support known or conventionaloperating features that are not described in detail herein. Thecomponents (202, 204, 206, 208, and 210) are communicatively coupled viaa local interface 212. The local interface 212 may be, for example, butnot limited to, one or more buses or other wired or wirelessconnections, as is known in the art. The local interface 212 may haveadditional elements, which are omitted for simplicity, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, amongmany others, to enable communications. Further, the local interface 212may include address, control, and/or data connections to enableappropriate communications among the aforementioned components.

The processor 202 is a hardware device for executing softwareinstructions. The processor 202 may be any custom made or commerciallyavailable processor, a Central Processing Unit (CPU), an auxiliaryprocessor among several processors associated with the server 200, asemiconductor-based microprocessor (in the form of a microchip orchipset), or generally any device for executing software instructions.When the server 200 is in operation, the processor 202 is configured toexecute software stored within the memory 210, to communicate data toand from the memory 210, and to generally control operations of theserver 200 pursuant to the software instructions. The I/O interfaces 204may be used to receive user input from and/or for providing systemoutput to one or more devices or components.

The network interface 206 may be used to enable the server 200 tocommunicate on a network, such as the Internet 104. The networkinterface 206 may include, for example, an Ethernet card or adapter or aWireless Local Area Network (WLAN) card or adapter. The networkinterface 206 may include address, control, and/or data connections toenable appropriate communications on the network. A data store 208 maybe used to store data. The data store 208 may include any of volatilememory elements (e.g., random access memory (RAM, such as DRAM, SRAM,SDRAM, and the like)), nonvolatile memory elements (e.g., ROM, harddrive, tape, CDROM, and the like), and combinations thereof.

Moreover, the data store 208 may incorporate electronic, magnetic,optical, and/or other types of storage media. In one example, the datastore 208 may be located internal to the server 200, such as, forexample, an internal hard drive connected to the local interface 212 inthe server 200. Additionally, in another embodiment, the data store 208may be located external to the server 200 such as, for example, anexternal hard drive connected to the I/O interfaces 204 (e.g., SCSI orUSB connection). In a further embodiment, the data store 208 may beconnected to the server 200 through a network, such as, for example, anetwork-attached file server.

The memory 210 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, tape, CDROM, etc.), andcombinations thereof. Moreover, the memory 210 may incorporateelectronic, magnetic, optical, and/or other types of storage media. Notethat the memory 210 may have a distributed architecture, where variouscomponents are situated remotely from one another but can be accessed bythe processor 202. The software in memory 210 may include one or moresoftware programs, each of which includes an ordered listing ofexecutable instructions for implementing logical functions. The softwarein the memory 210 includes a suitable Operating System (O/S) 214 and oneor more programs 216. The operating system 214 essentially controls theexecution of other computer programs, such as the one or more programs216, and provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices. The one or more programs 216 may be configured to implementthe various processes, algorithms, methods, techniques, etc. describedherein.

§ 4.0 Example User Device Architecture

FIG. 5 is a block diagram of a user device 300, which may be used withthe cloud-based system 100 or the like. Specifically, the user device300 can form a device used by one of the users 102, and this may includecommon devices such as laptops, smartphones, tablets, netbooks, personaldigital assistants, MP3 players, cell phones, e-book readers, IoTdevices, servers, desktops, printers, televisions, streaming mediadevices, and the like. The user device 300 can be a digital device that,in terms of hardware architecture, generally includes a processor 302,I/O interfaces 304, a network interface 306, a data store 308, andmemory 310. It should be appreciated by those of ordinary skill in theart that FIG. 5 depicts the user device 300 in an oversimplified manner,and a practical embodiment may include additional components andsuitably configured processing logic to support known or conventionaloperating features that are not described in detail herein. Thecomponents (302, 304, 306, 308, and 302) are communicatively coupled viaa local interface 312. The local interface 312 can be, for example, butnot limited to, one or more buses or other wired or wirelessconnections, as is known in the art. The local interface 312 can haveadditional elements, which are omitted for simplicity, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, amongmany others, to enable communications. Further, the local interface 312may include address, control, and/or data connections to enableappropriate communications among the aforementioned components.

The processor 302 is a hardware device for executing softwareinstructions. The processor 302 can be any custom made or commerciallyavailable processor, a CPU, an auxiliary processor among severalprocessors associated with the user device 300, a semiconductor-basedmicroprocessor (in the form of a microchip or chipset), or generally anydevice for executing software instructions. When the user device 300 isin operation, the processor 302 is configured to execute software storedwithin the memory 310, to communicate data to and from the memory 310,and to generally control operations of the user device 300 pursuant tothe software instructions. In an embodiment, the processor 302 mayinclude a mobile optimized processor such as optimized for powerconsumption and mobile applications. The I/O interfaces 304 can be usedto receive user input from and/or for providing system output. Userinput can be provided via, for example, a keypad, a touch screen, ascroll ball, a scroll bar, buttons, a barcode scanner, and the like.System output can be provided via a display device such as a LiquidCrystal Display (LCD), touch screen, and the like.

The network interface 306 enables wireless communication to an externalaccess device or network. Any number of suitable wireless datacommunication protocols, techniques, or methodologies can be supportedby the network interface 306, including any protocols for wirelesscommunication. The data store 308 may be used to store data. The datastore 308 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, and the like)),nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, and thelike), and combinations thereof. Moreover, the data store 308 mayincorporate electronic, magnetic, optical, and/or other types of storagemedia.

The memory 310 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, etc.), and combinations thereof.Moreover, the memory 310 may incorporate electronic, magnetic, optical,and/or other types of storage media. Note that the memory 310 may have adistributed architecture, where various components are situated remotelyfrom one another but can be accessed by the processor 302. The softwarein memory 310 can include one or more software programs, each of whichincludes an ordered listing of executable instructions for implementinglogical functions. In the example of FIG. 3, the software in the memory310 includes a suitable operating system 314 and programs 316. Theoperating system 314 essentially controls the execution of othercomputer programs and provides scheduling, input-output control, fileand data management, memory management, and communication control andrelated services. The programs 316 may include various applications,add-ons, etc. configured to provide end user functionality with the userdevice 300. For example, example programs 316 may include, but notlimited to, a web browser, social networking applications, streamingmedia applications, games, mapping and location applications, electronicmail applications, financial applications, and the like. In a typicalexample, the end-user typically uses one or more of the programs 316along with a network such as the cloud-based system 100.

§ 5.0 Zero Trust Network Access Using the Cloud-Based System

FIG. 6 is a network diagram of a Zero Trust Network Access (ZTNA)application utilizing the cloud-based system 100. For ZTNA, thecloud-based system 100 can dynamically create a connection through asecure tunnel between an endpoint (e.g., users 102A, 102B) that areremote and an on-premises connector 400 that is either located in cloudfile shares and applications 402 and/or in an enterprise network 410that includes enterprise file shares and applications 404. Theconnection between the cloud-based system 100 and on-premises connector400 is dynamic, on-demand, and orchestrated by the cloud-based system100. A key feature is its security at the edge—there is no need to punchany holes in the existing on-premises firewall. The connector 400 insidethe enterprise (on-premises) “dials out” and connects to the cloud-basedsystem 100 as if too were an endpoint. This on-demand dial-outcapability and tunneling authenticated traffic back to the enterprise isa key differentiator for ZTNA. Also, this functionality can beimplemented in part by the application 350 on the user device 300. Also,the applications 402, 404 can include B2B applications. Note, thedifference between the applications 402, 404 is the applications 402 arehosted in the cloud, whereas the applications 404 are hosted on theenterprise network 410. The B2B service described herein contemplatesuse with either or both of the applications 402, 404.

The paradigm of virtual private access systems and methods is to giveusers network access to get to an application and/or file share, not tothe entire network. If a user is not authorized to get the application,the user should not be able even to see that it exists, much less accessit. The virtual private access systems and methods provide an approachto deliver secure access by decoupling applications 402, 404 from thenetwork, instead of providing access with a connector 400, in front ofthe applications 402, 404, an application on the user device 300, acentral authority 152 to push policy, and the cloud-based system 100 tostitch the applications 402, 404 and the software connectors 400together, on a per-user, per-application basis.

With the virtual private access, users can only see the specificapplications 402, 404 allowed by the central authority 152. Everythingelse is “invisible” or “dark” to them. Because the virtual privateaccess separates the application from the network, the physical locationof the application 402, 404 becomes irrelevant—if applications 402, 404are located in more than one place, the user is automatically directedto the instance that will give them the best performance. The virtualprivate access also dramatically reduces configuration complexity, suchas policies/firewalls in the data centers. Enterprises can, for example,move applications to Amazon Web Services or Microsoft Azure, and takeadvantage of the elasticity of the cloud, making private, internalapplications behave just like the marketing leading enterpriseapplications. Advantageously, there is no hardware to buy or deploybecause the virtual private access is a service offering to end-usersand enterprises.

§ 6.0 Digital Experience Monitoring

FIG. 7 is a network diagram of the cloud-based system 100 in anapplication of digital experience monitoring. Here, the cloud-basedsystem 100 providing security as a service as well as ZTNA, can also beused to provide real-time, continuous digital experience monitoring, asopposed to conventional approaches (synthetic probes). A key aspect ofthe architecture of the cloud-based system 100 is the inline monitoring.This means data is accessible in real-time for individual users fromend-to-end. As described herein, digital experience monitoring caninclude monitoring, analyzing, and improving the digital userexperience.

The cloud-based system 100 connects users 102 at the locations 110, 112,118 to the applications 402, 404, the Internet 104, the cloud services106, etc. The inline, end-to-end visibility of all users enables digitalexperience monitoring. The cloud-based system 100 can monitor, diagnose,generate alerts, and perform remedial actions with respect to networkendpoints, network components, network links, etc. The network endpointscan include servers, virtual machines, containers, storage systems, oranything with an IP address, including the Internet of Things (IoT),cloud, and wireless endpoints. With these components, these networkendpoints can be monitored directly in combination with a networkperspective. Thus, the cloud-based system 100 provides a uniquearchitecture that can enable digital experience monitoring, networkapplication monitoring, infrastructure component interactions, etc. Ofnote, these various monitoring aspects require no additionalcomponents—the cloud-based system 100 leverages the existinginfrastructure to provide this service.

Again, digital experience monitoring includes the capture of data abouthow end-to-end application availability, latency, and quality appear tothe end user from a network perspective. This is limited to the networktraffic visibility and not within components, such as what applicationperformance monitoring can accomplish. Networked application monitoringprovides the speed and overall quality of networked application deliveryto the user in support of key business activities. Infrastructurecomponent interactions include a focus on infrastructure components asthey interact via the network, as well as the network delivery ofservices or applications. This includes the ability to provide networkpath analytics.

The cloud-based system 100 can enable real-time performance andbehaviors for troubleshooting in the current state of the environment,historical performance and behaviors to understand what occurred or whatis trending over time, predictive behaviors by leveraging analyticstechnologies to distill and create actionable items from the largedataset collected across the various data sources, and the like. Thecloud-based system 100 includes the ability to directly ingest any ofthe following data sources network device-generated health data, networkdevice-generated traffic data, including flow-based data sourcesinclusive of NetFlow and IPFIX, raw network packet analysis to identifyapplication types and performance characteristics, HTTP request metrics,etc. The cloud-based system 100 can operate at 10 gigabits (10G)Ethernet and higher at full line rate and support a rate of 100,000 ormore flows per second or higher.

The applications 402, 404 can include enterprise applications, Office365, Salesforce, Skype, Google apps, internal applications, etc. Theseare critical business applications where user experience is important.The objective here is to collect various data points so that userexperience can be quantified for a particular user, at a particulartime, for purposes of analyzing the experience as well as improving theexperience. In an embodiment, the monitored data can be from differentcategories, including application-related, network-related,device-related (also can be referred to as endpoint-related),protocol-related, etc. Data can be collected at the application 350 orthe cloud edge to quantify user experience for specific applications,i.e., the application-related and device-related data. The cloud-basedsystem 100 can further collect the network-related and theprotocol-related data (e.g., Domain Name System (DNS) response time).

Application-Related Data

Page Load Time Redirect count (#) Page Response Time Throughput (bps)Document Object Model Total size (bytes) (DOM) Load Time TotalDownloaded bytes Page error count (#) App availability (%) Page elementcount by category (#)

Network-Related Data

HTTP Request metrics Bandwidth Server response time Jitter Ping packetloss (%) Trace Route Ping round trip DNS lookup trace Packet loss (%)GRE/IPSec tunnel monitoring Latency MTU and bandwidth measurements

Device-Related Data (Endpoint-Related Data)

System details Network (config) Central Processing Unit (CPU) DiskMemory (RAM) Processes Network (interfaces) Applications

Metrics could be combined. For example, device health can be based on acombination of CPU, memory, etc. Network health could be a combinationof Wi-Fi/LAN connection health, latency, etc. Application health couldbe a combination of response time, page loads, etc. The cloud-basedsystem 100 can generate service health as a combination of CPU, memory,and the load time of the service while processing a user's request. Thenetwork health could be based on the number of network path(s), latency,packet loss, etc.

The lightweight connector 400 can also generate similar metrics for theapplications 402, 404. In an embodiment, the metrics can be collectedwhile a user is accessing specific applications that user experience isdesired for monitoring. In another embodiment, the metrics can beenriched by triggering synthetic measurements in the context of aninline transaction by the application 350 or cloud edge. The metrics canbe tagged with metadata (user, time, app, etc.) and sent to a loggingand analytics service for aggregation, analysis, and reporting. Further,network administrators can get UEX reports from the cloud-based system100. Due to the inline nature and the fact the cloud-based system 100 isan overlay (in-between users and services/applications), the cloud-basedsystem 100 enables the ability to capture user experience metric datacontinuously and to log such data historically. As such, a networkadministrator can have a long-term detailed view of the network andassociated user experience.

§ 7.0 Cloud Tunnel

FIG. 8 is a network diagram of the cloud-based system 100 with variouscloud tunnels 500, labeled as cloud tunnels 500A, 500B, 500C, forforwarding traffic. FIGS. 9 and 10 are flow diagrams of a cloud tunnel500 illustrating a control channel (FIG. 9) and a data channel (FIG.10), with the tunnel illustrated between a client 510 and a server 520.The cloud tunnel 500 is a lightweight tunnel that is configured toforward traffic between the client 510 and the server 520. The presentdisclosure focuses on the specific mechanisms used in the cloud tunnel500 between two points, namely the client 510 and the server 520. Thoseskilled in the art will recognize the cloud tunnel 500 can be used withthe cloud-based system 100 as an example use case, and other uses arecontemplated. That is, the client 510 and the server 520 are justendpoint devices that support the exchange of data traffic and controltraffic for the tunnel 500. For description, the server 520 can bereferred to as a local node and the client 510 as a remote node, wherethe tunnel operates between the local and remote nodes.

In an embodiment, the cloud-based system 100 can use the cloud tunnel500 to forward traffic to the enforcement nodes 150, such as from a userdevice 300 with the application 350, from a branch office/remotelocation 118, etc. FIG. 8 illustrates three example use cases for thecloud tunnel 500 with the cloud-based system 100, and other uses arealso contemplated. In a first use case, a cloud tunnel 500A is formedbetween a user device 300, such as with the application 350, and anenforcement node 150-1. For example, when a user 102 associated with theuser device 300 connects to a network, the application 350 can establishthe cloud tunnel 500A to the closest or best enforcement node 150-1, andforward the traffic through the cloud tunnel 500A so that theenforcement node 150-1 can apply the appropriate security and accesspolicies. Here, the cloud tunnel 500A supports a single user 102,associated with the user device 300.

In a second use case, a cloud tunnel 500B is formed between a VirtualNetwork Function (VNF) 502 or some other device at a remote location118A and an enforcement node 150-2. Here, the VNF 502 is used to forwardtraffic from any user 102 at the remote location 118A to the enforcementnode 150-2. In a third use case, a cloud tunnel 110C is formed betweenan on-premises enforcement node, referred to as an Edge Connector (EC)150A, and an enforcement node 150-N. The edge connector 150A can belocated at a branch office 118A or the like. In some embodiments, theedge connector 150A can be an enforcement node 150 in the cloud-basedsystem 100 but located on-premises with a tenant. Here, in the secondand third use cases, the cloud tunnels 500B, 500C support multiple users102.

There can be two versions of the cloud tunnel 500, referred to a tunnel1 and tunnel 2. The tunnel 1 can only support Web protocols as an HTTPconnect tunnel operating on a TCP streams. That is, the tunnel 1 cansend all proxy-aware traffic or port 80/443 traffic to the enforcementnode 150, depending on the forwarding profile configuration. This can beperformed via CONNECT requests, similar to a traditional proxy.

The tunnel 2 can support multiple ports and protocols, extending beyondonly web protocols. As described herein, the cloud tunnels 500 are thetunnel 2. In all of the use cases, the cloud tunnel 500 enables eachuser device 300 to redirect traffic destined to all ports and protocolsto a corresponding enforcement node 150. Note, the cloud-based system100 can include load balancing functionality to spread the cloud tunnels500 from a single source IP address. The cloud tunnel 500 supportsdevice logging for all traffic, firewall, etc., such as in the storagecluster 156. The cloud tunnel 500 utilizes encryption, such as via TLSor DTLS, to tunnel packets between the two points, namely the client 510and the server 520. As described herein, the client 510 can be the userdevice 300, the VNF 502, and/or the edge connector 150A, and the server520 can be the enforcement node 150. Again, other devices arecontemplated with the cloud tunnel 500.

The cloud tunnel 500 can use a Network Address Translation (NAT) devicethat does not require a different egress IP for each device's 300separate sessions. Again, the cloud tunnel 500 has a tunnelingarchitecture that uses DTLS or TLS to send packets to the cloud-basedsystem 100. Because of this, the cloud tunnel 500 is capable of sendingtraffic from all ports and protocols.

Thus, the cloud tunnel 500 provides complete protection for a singleuser 102, via the application 350, as well as for multiple users atremote locations 118, including multiple security functions such ascloud firewall, cloud IPS, etc. The cloud tunnel 500 includes user-levelgranularity of the traffic, enabling different users 102 on the samecloud tunnel 500 for the enforcement nodes 150 to provide user-basedgranular policy and visibility. In addition to user-level granularity,the cloud tunnel 500 can provide application-level granularity, such asby mapping mobile applications (e.g., Facebook, Gmail, etc.) to traffic,allowing for app-based granular policies.

FIGS. 9 and 10 illustrate the two communication channels, namely acontrol channel 530 and a data channel 540, between the client 510 andthe server 520. Together, these two communication channels 530, 540 formthe cloud tunnel 500. In an embodiment, the control channel 530 can bean encrypted TLS connection or SSL connection, and the control channel530 is used for device and/or user authentication and other controlmessages. In an embodiment, the data channel 540 can be an encryptedDTLS or TLS connection, i.e., the data channel can be one or more DTLSor TLS connections for the transmit and receive of user IP packets.There can be multiple data channels 540 associated with the same controlchannel 530. The data channel 540 can be authenticated using a SessionIdentifier (ID) from the control channel 530.

Of note, the control channel 530 always uses TLS because some locations(e.g., the remote location 118A, the branch office 118B, otherenterprises, hotspots, etc.) can block UDP port 443, preventing DTLS.Whereas TLS is widely used and not typically blocked. The data channel540 preferably uses DTLS, if it is available, i.e., not blocked on theclient 510. If it is blocked, the data channel 540 can use TLS instead.For example, DTLS is the primary protocol for the data channel 540 withTLS used as a fallback over TCP port 443 if DTLS is unavailable, namelyif UDP port 443 is blocked at the client 510.

In FIG. 9, the control channel 530 is illustrated with exchanges betweenthe client 510 and the server 520. Again, the control channel 530includes TLS encryption, which is established through a setup orhandshake between the client 510 and the server 520 (step 550-1). Anexample of a handshake is illustrated in FIG. 11. The client 510 cansend its version of the tunnel 500 to the server 520 (step 550-2) towhich the server 520 can acknowledge (step 550-3). For example, theversion of the tunnel can include a simple version number or otherindication, as well as an indication of whether the client 510 supportsDTLS for the data channel 540. Again, the control channel 530 is fixedwith TLS or SSL, but the data channel 540 can be either DTLS or TLS.

The client 510 can perform device authentication (step 550-4), and theserver 520 can acknowledge the device authentication (step 550-5). Theclient 510 can perform user authentication (step 550-6), and the server520 can acknowledge the user authentication (step 550-7). Note, thedevice authentication includes authenticating the user device 300, suchas via the application 350, the VNF 502, the edge connector 150A, etc.The user authentication includes authenticating the users 102 associatedwith the user devices 300. Note, in an embodiment, the client 510 is thesole device 300, and here the user authentication can be for the user102 associated with the client 510, and the device authentication can befor the user device 300 with the application 350. In another embodiment,the client 510 can have multiple user devices 300 and correspondingusers 102 associated with it. Here, the device authentication can be forthe VNF 502, the edge connector 150A, etc., and the user authenticationcan be for each user device 300 and corresponding user 102, and theclient 510 and the server 520 can have a unique identifier for each userdevice 300, for user-level identification.

The device authentication acknowledgment can include a sessionidentifier (ID) that is used to bind the control channel 530 with one ormore data channels 540. The user authentication can be based on a useridentifier (ID) that is unique to each user 102. The client 510 canperiodically provide keep alive packets (step 550-8), and the server 520can respond with keep alive acknowledgment packets (step 550-9). Theclient 510 and the server 520 can use the keep alive packets or messagesto maintain the control channel 530. Also, the client 510 and the server520 can exchange other relevant data over the control channel 530, suchas metadata, which identifies an application for a user 102, locationinformation for a user device 300, etc.

In FIG. 10, similar to FIG. 9, the data channel 540 is illustrated withexchanges between the client 510 and the server 520. Again, the datachannel 540 includes TLS or DTLS encryption, which is establishedthrough a setup or handshake between the client 510 and the server 520(step 560-1). An example of a handshake is illustrated in FIG. 11. Note,the determination of whether to use TLS or DTLS is based on the sessionID, which is part of the device authentication acknowledgment, and whichis provided over the data channel 540 (steps 560-2, 560-3). Here, theclient 510 has told the server 520 its capabilities, and the session IDreflects what the server 520 has chosen, namely TLS or DTLS, based onthe client's 510 capabilities. In an embodiment, the server 520 choosesDTLS if the client 510 supports it, i.e., if UDP port 443 is notblocked, otherwise the server 520 chooses TLS. Accordingly, the controlchannel 530 is established before the data channel 540. The data channel540 can be authenticated based on the session ID from the controlchannel 530.

The data channel 540 includes the exchange of data packets between theclient 510 and the server 520 (step 560-4). The data packets include anidentifier such as the session ID and a user ID for the associated user102. Additionally, the data channel 540 can include keep alive packetsbetween the client 510 and the server 520 (steps 560-5, 560-6).

The cloud tunnel 500 can support load balancing functionality betweenthe client 510 and the server 520. The server 520 can be in a cluster,i.e., multiple servers 200. For example, the server 520 can be anenforcement node 150 cluster in the cloud-based system 100. Becausethere can be multiple data channels 540 for a single control channel530, it is possible to have the multiple data channels 540, in a singlecloud tunnel 500, connected to different physical servers 200 in acluster. Thus, the cloud-based system 100 can include load balancingfunctionality to spread the cloud tunnels 500 from a single source IPaddress, i.e., the client 510.

Also, the use of DTLS for the data channels 540 allows the user devices300 to switch networks without potentially impacting the traffic goingthrough the tunnel 500. For example, a large file download couldcontinue uninterrupted when a user device 300 moves from Wi-Fi tomobile, etc. Here, the application 350 can add some proprietary data tothe DTLS client-hello servername extension. That proprietary data helpsa load balancer balance the new DTLS connection to the same server 200in a cluster where the connection prior to network change was beingprocessed. So, a newly established DTLS connection with different IPaddress (due to network change) can be used to tunnel packets of thelarge file download that was started before the network change. Also,some mobile carriers use different IP addresses for TCP/TLS (controlchannel) and UDP/DTLS (data channel) flows. The data in DTLSclient-hello helps the load balancer balance the control and dataconnection to the same server 200 in the cluster.

§ 8.0 Traceroute

Traceroute can be based on Internet Control Message Protocol (ICMP),TCP, User Datagram Protocol (UDP), etc. For example, a traceroute basedon ICMP provides all hops on the network. TCP and UDP are also supportedby most clients, if ICMP is blocked. The response from the tracerouteprovides a holistic view of the network with packet loss details andlatency details. FIG. 11 is a network diagram of a traceroute between auser 102 and a destination 640 with no tunnel in between. Here, the user102 (via a user device 300) connects to an access point 600, whichconnects to the destination 640 via routers 602A-602D and a switch 604.The traceroute includes transmitting a request packet from the user 102to the destination 640 (with an address of a.b.c.d) via the access point600, the routers 602, and the switch 604. Each of these intermediatedevices 600, 602, 604 process the request packet and the enforcementnode 150 sends a response packet back to the user 102, which is alsoprocessed by the intermediate devices 600, 602, 604. Accordingly, allhops in the network are visible.

FIG. 12 is a network diagram of a traceroute between a user 102 and thedestination 640 with an opaque tunnel 610 between a tunnel client 510and a tunnel server 520. The opaque tunnel 610 can be the tunnel 500 aswell as a GRE, IPsec, VPN, etc. The opaque tunnel 610 is referred to asopaque because there is no visibility into the tunnel. The traceroute inFIG. 12, based on ICMP, TCP, UDP, etc., provides visibility of the hopsbefore and after the opaque tunnel 610, but does not provide visibilityin the opaque tunnel 610. There are no details about packet loss orlatency while tunneled transmission. Also, the opaque tunnel 610 can bereferred to as an overlay tunnel.

Traceroute includes a series of packets that are exchanged from a probeinitiator along a path. Each trace packet includes an increasing TTLvalue. When a node along the path receives a trace packet where the TTLexpires, it sends a response. Based on all of the responses, it ispossible for the probe initiator (e.g., the client) to determine thenetwork hops, the latency at each hop, packet loss, and other details.Again, the traceroute can be an MTR, which also includes PINGfunctionality. Again, MTR is used to traceroute the destination to showthe latency, packet loss, and hop information between an initiator anddestination. It helps to understand the network status and diagnosenetwork issues.

In an embodiment, MTR is implemented on the user device 300, such asthrough the application 350, and on the tunnel server 520 and/or theenforcement node 150. As is described herein, there is a requirement toimplement probes at two points in the service path—at the client and atthe tunnel server 520 and/or the enforcement node 150. The MTRimplementation can support ICMP, UDP, and/or TCP. For ICMP, two socketsare used to send and receive probes, and the ICMP sequence number inreply messages are used to match ICMP request messages. For UDP, one UDPsocket is created to send UDP probes, and one ICMP socket is created toreceive ICMP error messages. For TCP, one raw socket is created to sendTCP probes, and one ICMP socket is created to receive ICMP errormessages, and the TCP socket is also used to receive SYN-ACK/RST fromthe destination. The foregoing functionality can be performed by theapplication 350 on the user device 300 and a tracing service on theenforcement node 150. SYN=Synchronize, ACK=Acknowledgment, andRST=Reset.

§ 8.1 Detecting Opaque Tunnel

FIG. 13 is a flowchart of a process 650 for detecting a tunnel 500, 610between a user device 300 and a destination. The process 650 isdescribed with reference to the network in FIG. 12 with actions at theuser device 300, the intermediate devices 600, 602, 604, and the tunnelserver 520. Also, note that while the enforcement node 150 and thetunnel server 520 are illustrated as separate devices, it is alsopossible that these are combined in the same device. Also, actions atthe user device 300 (client) can be performed via the application 350executed thereon. The tunnel server 520 can be a proxy or transparentproxy.

The process 650 includes the client sending a trace packet for thedestination (e.g., the node 150 with an address of a.b.c.d) with aSignature-A (step 651). Note, the client (e.g., the user device 300)does not know if there is a tunnel or not between the destination anditself. The purpose of the Signature-A is for any tunnel server 520 todetect this trace packet and provide tunnel details, i.e., to allow theclient to detect the tunnel. The Signature-A can be any encrypted datafor security.

The process 650 further includes the tunnel server detecting theSignature-A as a valid signature and intercepting the trace packet (step652). In FIG. 12, even though the tunnel server 520 is not thedestination, it intercepts the trace packet because of the presence ofthe Signature-A and responds. Namely, the tunnel server responds to thetrace packet with tunnel info (step 653). The client receives the traceresponse from the tunnel server (instead of the destination) and isinformed about the tunnel, and can take appropriate action (step 654).The tunnel info can include IP address, tunnel type, protocol, etc. Asdescribed herein, appropriate action includes determining a trace viadifferent legs to account for the tunnel. Also, as described herein, aleg is a segment of the network between the client and the destination.Without a tunnel, there is a single leg between the client and thedestination. With a tunnel, there is a plurality of legs with at leastone leg being the tunnel itself.

If there is a transparent proxy present with an overlay tunnel to itfrom the client, the client sends traceroute probes with a signature todetect the presence of the proxy. When the packets traverse through theproxy, it scans for the signature in the payload, which can be encryptedusing a shared key that can be rotated constantly. If the signaturematches, the proxy identifies this as a probe generated by a trustedclient and identifies itself as a proxy by responding to the probe withan encrypted signature. On receiving the probe response, the clientwould be able to identify the proxy in the path and request it to findthe hops through the overlay tunnel. The request to the proxy can beperformed out of band.

§ 8.2 Collecting Network Details Including a Tunnel

FIG. 14 is a flowchart of a process 660 for collecting network detailsin a trace where there is an opaque tunnel. The process 660 is describedwith reference to the network in FIG. 12 with actions at the user device300, the intermediate devices 600, 602, 604, and the tunnel server 520.Further, the process 660 can be used with the process 660. Also, whiledescribed with reference to the enforcement node 150 as the destination,the process 660 contemplates operation with any type of computingdevice. Also, note that while the enforcement node 150 and the tunnelserver 520 are illustrated as separate devices, it is also possible thatthese are combined in the same device. Also, actions at the user device300 (client) can be performed via the application 350 executed thereon.

Once an opaque tunnel is detected, the process 660 is used to collectdetails of the service path between the client and the destination. Theprocess 660 includes, responsive to detection of a tunnel, dividing thenetwork from the client to the destination into a plurality of legs(step 661). A trace is performed separately on all of the plurality oflegs (step 662), and the results of the trace on all of the plurality oflegs are aggregated to provide a holistic view of the network (step663).

The objective in segmenting the network into different legs is toprovide visibility with the tunnel. Specifically, a trace is performedin the tunnel, such as via the tunnel server which is performing aso-called “reverse traceroute.” Here, the tunnel server is sending tracepackets through the tunnel without tunnel encapsulation so that detailsof the trace can be obtained in the opaque tunnel. These details arecombined with traces from the other legs to provide full visibility.

For the example of FIG. 12, once the client (user device 300) knowsabout tunnel, the network can be divided into three segments:

Leg-1: From the user device 300 to an egress router 630,

Leg-2: From the tunnel client 510 to the tunnel server 520 (i.e., theopaque tunnel 610), and

Leg-3: From the tunnel server 520 to the destination (node 150).

For the Leg-1, the trace can be performed as normal.

For the Leg-2, the trace is performed between the egress router 630 andthe tunnel server 520. This is the reverse traceroute where the tunnel610 is traced by the tunnel server. In an embodiment, the client,knowing there is an opaque tunnel based on the signature used in theprocess 650, requests the tunnel server trace the tunnel. That is, theclient sends a request for tracing by the tunnel server to the tunnelclient, i.e., a reverse trace. The tunnel server performs the reversetrace, collects the results and forwards them to the client.

For the Leg-3, either the client can send a trace packet without thesignature to trace the Leg-3 or the client can request the tunnel serverperform a trace to the destination on its behalf. If the trace packet issent from the client without the signature, the results will includedetails from Legs 1 and 2, which can be subtracted out since the resultsfrom Legs 1 and 2 are also separately obtained. Finally, the client canprocess all of the results from the three legs to present a holisticview of the network. Note, Leg-2 and Leg-3 go habd in hand—either youhave both or none. If there is none, then the client only has one leg tothe destination.

The foregoing assumes the tunnel client 510 is on the public Internetand reachable from the tunnel server 520, i.e., the outside world canconnect to the tunnel client 510. However, most tunnel clients 510 areon an internal network behind a firewall, making it a problem for thetunnel server 520 to reverse trace to the tunnel client 510. Thus, thereare additional steps in this scenario.

Consider the issue of the tunnel client 510 being behind a firewall;there is a need to modify the network segments as follows:

Leg-1: From the user device 300 to an egress router 630,

Leg-2: From the egress router 630 to the tunnel server 520, and

Leg-3: From the tunnel server 520 to the destination.

As described herein, the egress router 630 is typically a router at anedge of a customer's network with a public IP address. The followingdescribes the trace in each of these legs. For the Leg-3, the client cansend the trace packet without the signature or request the tunnel server520 to perform this leg on its behalf, i.e., the same as describedabove.

For the Leg-2, the following steps are needed, note these are asdescribed above except the target is the egress router 630. The tunnelserver 520 is performing a reverse trace based on accepting a requestfrom the client, but the reverse trace is from the tunnel server 520 tothe egress router 630. The tunnel server 520 provides the results to theclient as before.

For the Leg-1, the client sends a trace packet to the egress router 630.And as before, finally, the client aggregates all three legs to presenta holistic view of the network.

For the Leg-1, there are two possibilities for what can happen to thetrace packet from the client to the egress router. For a case-1, thetunnel client 510 can route the trace packet into the opaque tunnel 610.For a case-2, the tunnel client 510 does not route the trace packet intothe opaque tunnel 610, i.e., bypasses it. For the case-2, this yieldsthe trace to the egress router 630 data. However, for the case-1, thisprovides the wrong network path, namely from the client to the tunnelclient 510 to the tunnel server 520 to the Internet to the egress router630. That is, the trace packet echoes from the tunnel server 520providing the wrong network path. There is a need for the client todetect this wrong network path.

To detect the wrong path for the Leg-1, the client can be configured toinsert another signature, Signature-B, in the trace packet for theegress router 630. The objective is for the trace packet to reach theegress router 630 for a response. The purpose of this Signature-B is forthe tunnel server 520 to detect it and provide a flag in the response.If the client gets a response to this trace packet with the flagtherein, the client knows the trace went on the wrong network path,i.e., through the tunnel 610 to the tunnel server 520. When this isdetected, IT must reconfigure the tunnel client 510 to bypass the tunnel610 for packets destined to the egress router 630. Of note, the use ofthe terms Signature-A and Signature-B is solely meant to differentiatethese as different signatures for different purposes.

As described herein, the present disclosure includes various traces ofdifferent legs of a service path, such as using MTR, and having theclient (or another device) aggregate the results. Of note, while theillustrated example embodiments describe the traces in order, thoseskilled in the art will appreciate any order is contemplated. Forexample, in some embodiments, the traces of Leg 1 are performed first,then Leg 2, etc. In other embodiments, the traces of Leg 2 are performedfirst, etc. Finally, the traces may be performed concurrently or atabout the same time.

In an embodiment, the tunnel client 510 can be a tunnel originating fromthe application 350 and the egress router 630 can represent the publicfacing side of the network from where location tunnels (GRE/IPSEC) willoriginate. Most cases will have the user device 300 on a private IPtalking to the outside world via a router or a Wi-Fi Access Point (AP)that is connected to an egress router 630 that has a public IP. The caseof a tunnel client 510 having a public IP is rare and could happen whenthere is a device on cellular network. From the point of the enforcementnode 150, it always traces the Leg 2 path from itself to the public IPthe client comes out with. It does not care if it is an egress router ora tunnel-client end point that is on the public IP.

§ 8.3 Example Operation

FIG. 15 is a flow diagram illustrating actions between the client (userdevice 300), the tunnel client 510, the egress router 630, the tunnelserver 520, and the destination 640 in an example operation of theprocesses 650, 660. Note, the processes 650, 660 can be orchestrated bythe user device 300 (client) via the application 350. The client sends atrace packet to the destination with the Signature-A as described in theprocess 650. If the response comes back with no tunnel info in theresponse, then the full and accurate service path has been traced andthe traceroute is complete. If there is tunnel info, the client knowsthere is the tunnel 610 and moves to the process 620.

In order to collect a full network path, first the client needs todetect if there is a tunnel on the path. Again, this is achieved by theclient inserting a signature in a packet. The packet is intercepted bythe tunnel server 520 and it will respond with tunnel information liketype, IP, etc. Once the client notices the tunnel on the path, it willrun the multi-segment approach in the process 660 to detect the fullservice path.

Next, the client fetches the egress IP using the restful API. The clientassumes three network segments—Leg-1: Client to Egress, Leg-2: Egress toTunnel Server, and Leg-3: Tunnel Server to Destination. The clientperforms the trace of the Leg-3 either directly or by requesting thetunnel server to perform it and collect information. The client performsthe trace of Leg-2 by requesting the tunnel server perform the reversetrace. The client also sends a trace packet to the egress router 630with the Signature-B. If there is no tunnel flag in the response, theclient has the full and accurate Leg-1 information. If there is thetunnel flag in the response, there is a misconfiguration presented tothe user.

Finally, the client aggregates all three legs and consumes the data. Thetunnel server 520 can host a tracing service that will accept tracingrequests from clients such as via a restful API call, an HTTP Post call,etc. This service will perform standard network tracing, collect thedata and respond to clients. The resultant data can be displayed andused in different ways.

§ 9.0 Detection of Network Hops and Latency Through an Opaque Tunnel andDetection Misconfiguration of Tunnels

FIG. 16 is a flowchart of a process 670 for detection of network hopsand latency through an opaque tunnel and detection misconfiguration oftunnels. The process 670 is described with reference to the user device300, i.e., the client. The process 670 can be implemented as a methodthat includes steps, via the user device 300 configured to execute thesteps, and via a non-transitory computer-readable medium that includesinstructions that cause one or more processors to implement the steps.

The process 670 includes requesting a trace to a destination with asignature inserted into a trace packet (step 671); receiving a responseto the trace packet (step 672); when the response does not includetunnel info, providing details in the response to a service where thedetails include parameters associated with a service path between theclient and the destination (step 673); and when the response includestunnel info, segmenting the service path into a plurality of legs,causing a trace for each of the plurality of legs, and aggregatingdetails for each of the plurality of legs based on the causing (step674).

When the response includes tunnel info, a tunnel server is configured tointercept the trace packet responsive to detection of the signature, andwherein the tunnel server responds to the trace packet with the responsewith the tunnel info. The aggregating details includes aggregatingnetwork hops, packet drops, and latency for each of the plurality oflegs. The plurality of legs can include three legs. In an embodiment, afirst leg is between the client and a tunnel client, a second leg isbetween the tunnel client and a tunnel server, and a third leg isbetween the tunnel server and the destination. In another embodiment, afirst leg is between the client and an egress router, a second leg isbetween the egress router and a tunnel server, and a third leg isbetween the tunnel server and the destination.

The causing the trace for the plurality of legs can further includeincluding a second signature in a second trace packet to an egressrouter, and the process 670 can further include receiving a responsefrom the second trace packet; when the response does not include a flag,utilizing details from the response for a leg between the client and theegress router; and when the response includes the flag, determining amisconfiguration where the second trace packet was sent over a tunnel.At least one of the plurality of legs can include a reverse trace from atunnel server. The tunnel info can include a type of tunnel includingany of Generic Routing Encapsulation (GRE) and Internet Protocol (IP)Security (IPsec).

The process 670 helps detect the network hops, packet drops, and theirlatencies through tunnels like the GRE/IPsec or any other overlaytunnel. A typical network analyzer will not be able to find the hops,packet drops and their latency through individual routers thatconstitute the overlay tunnel as the probe traffic is encapsulatedthrough the tunnel and the whole tunnel looks like a single hop. Theprocess 670 enables a trace of the hops through the tunnel thus givingan insight into the hops inside the tunnel. The tracing of the path isdone by initiating the probes from the other side of the tunnel withoutencapsulating the packet, i.e., from the a destination 640 towards theclient which is called as “Reverse Traceroute” as described herein. Thisalso helps detect if the overlay tunnels are correctly configured sothat traffic bound to the internal network is not pulled into thetunnel.

§ 10.0 Detection of Latency, Packet Drops, and Network Hops Through aTCP Tunnel Using ICMP and UDP Probes

In another embodiment, the tunnel can include a TCP connection, i.e., aTCP-based tunnel or an exclusive TCP overlay tunnel. The presentdisclosure can trace this path to detect statistics such as hops, packetdrops, and latency through the exclusive TCP overlay tunnel using ICMPand UDP traffic. This approach leverages the approach in the process 670to find the hops through the tunnel using a protocol other than TCP forwhich the tunnel was built. This approach uses the routing in theopposite direction as the enforcement of the TCP check made at the endof the tunnel that the client owns. The destination 640 sends probesfrom its side of the tunnel without using any tunnel encapsulationtowards the client's egress router's IP.

Advantageously, this approach avoids using TCP-PINGs (use of TCP SYNs)from the client side towards the destination to avoid cases wherefirewall rules would flag issues thinking of it as an attack.

FIG. 17 is a flowchart of a process 680 for detection of latency, packetdrops, and network hops through a TCP tunnel using ICMP and UDP probes.The process 680 is described with reference to the destination 640. Theprocess 680 can be implemented as a method that includes steps, via theserver 200 configured to execute the steps, and via a non-transitorycomputer-readable medium that includes instructions that cause one ormore processors to implement the steps.

The process 680 includes receiving a request from a client to perform areverse trace (step 681); requesting a trace to an endpoint that is oneof an egress router and a tunnel client, wherein there is a tunnelbetween i) the destination and ii) the one of the egress router and thetunnel client (step 682); receiving a response to the trace (step 683);and sending details associated with the response to the client so thatthe client aggregates these details with details from one or moreadditional legs to provide an overall view of a service path between theclient and the destination (step 684).

The process 680 can further include receiving a trace packet from theclient with a signature included therein, wherein the signature isindicative of a request for tunnel info; and, responsive to detection ofthe signature, sending the tunnel info to the client in a response. Theprocess 680 can further include receiving a trace packet from the clientwith a signature included therein, wherein the signature is indicativeof a misconfiguration of a tunnel; and, responsive to detection of thesignature, sending a flag to the client in a response indicative of themisconfiguration.

The destination can be one of a tunnel server and a node in acloud-based system. The tunnel can utilize Transmission Control Protocol(TCP) and the trace to the endpoint utilizes a packet without tunnelencapsulation. The packet can utilize one of Internet Control MessageProtocol (ICMP) and User Datagram Protocol (UDP). The request can be viaa RESTful (Representational State Transfer) Application ProgrammingInterface (API) call from the client.

§ 11.0 Detection of Latency, Packet Drops, and Network Hops Through aTunnel by Tracing Hops Therein

As described above, the tunnel 610 is an opaque overlay making itdifficult to trace. The aforementioned approaches contemplate a reversetrace via unencapsulated packets. In an embodiment, the tunnel itselfmay be configured to perform the trace, such as via the cloud tunnel500. There are two techniques the tunnel 500 can use to perform thetrace inside the tunnel.

In a first approach, the tunnel 500 can be configured to identify probetraffic based on a predefined signature and inherits the IP TTL value ofthe probe packet. Note, as described herein, probe or probe trafficmeans traceroute packets. As the packet makes its way through the tunnelthe packet's TTL would expire triggering an ICMP “Time Exceeded” error.This error is propagated by the tunnel to the probe initiator (such asthe client) spoofing the IP address of the router that generated theerror.

In a second approach, the tunnel 500 itself can initiate tracerouteprobes towards the other end of the tunnel 500 by increasing the TTL inthe packets by one at a time. By tracing the path to the other end ofthe tunnel 500, the exact number of hops, packet drops, and latencyinside the tunnel 500 is determined. This information can be provided toany of the clients/applications via an API so that they know the measureof these stats that can be combined with the other traceroute stats toget a complete picture of the path the packet traverses. Thismeasurement can be initiated from both sides of the tunnel 500 to gaugeany changes in routing due to asymmetric routing.

FIG. 18 is a flowchart of a process 690 for detection of latency, packetdrops, and network hops through a tunnel by tracing hops therein. Theprocess 690 is described with reference to a node associated with thetunnel 500, i.e., either the tunnel client 510, the tunnel server 520,or the egress router 630. The process 690 can be implemented as a methodthat includes steps, via a processing device configured to execute thesteps, and via a non-transitory computer-readable medium that includesinstructions that cause one or more processors to implement the steps.

The process 690 includes receiving a request for a trace of the tunnelfrom a client (step 691); causing the trace inside the tunnel (step692); obtaining results of the trace inside the tunnel (step 693); andsending the results of the trace inside the tunnel to the client so thatthe client aggregates these details with details from one or moreadditional legs to provide an overall view of a service path between theclient and a destination (step 694).

The inside the tunnel can include identifying a packet with a predefinedsignature, analyzing a Time-to-Live (TTL) value in the packet, andsending a response to a probe initiator based on the TTL value. Theresponse can include an Internet Protocol (IP) address that was spoofedbased on a router where the TTL value expired.

The trace inside the tunnel can include sending trace packets to anotherend of the tunnel each having increasing Time-to-Live (TTL) values. Thetrace packets can be sent from both ends of the tunnel to determine anychanges in routing between directions.

The tunnel can include a data channel and a control channel each havingdifferent encryption. The encryption can be any of Transport LayerSecurity (TLS), Secure Sockets Layer (SSL), and Datagram Transport LayerSecurity (DTLS).

§ 12.0 Metric Computation for Traceroute Probes Using Cached Data toPrevent a Surge on Destination Servers

FIG. 19 is a network diagram illustrating a user 102 connected to anenforcement node 150 in a digital experience monitoring application. Ina practical embodiment, the cloud-based system 100 with the nodes 150 asproxies can be used to perform digital experience monitoring asdescribed herein. In such as system, there can be a lot of probes. Toprevent a surge of traffic to the destination 640, the presentdisclosure includes a cache approach where traceroute results are cachedon the proxy for a finite configurable time. For that time interval, allsubsequent probe requests are served out of the cache rather thansending a new set of probes per request. While one request is pending ona destination 640, any probe that arrives for the same destination canbe held in a queue and responded from the cache when the response forthe first probe arrives and is cached.

Specifically, if a lot of user devices 300 with the applications 350 areindependently probing the destination 640 there is a risk of throttlingof the probes at the destination 640 and the hops as well asblacklisting IP addresses of the tunnel server 520 or nodes 150 used toprobe the destination 640.

The enforcement node 150 is configured to probe the destination 640,i.e., the leg 3, on behalf of requesting clients. The enforcement node150 is also configured to probe the tunnel 500, 610 as described herein,i.e., leg 2, in a reverse trace. The present disclosure contemplates theenforcement node 150 caching results from these two legs and servingsubsequent requests from the cache for a predetermined amount of time.Each cache entry can include all hop IP addresses from the enforcementnode 150 to the destination 640 and from the enforcement node 150 to theegress router 630, packet loss, and latency for each probe sent. Note,some clients can share both legs 2 and 3 whereas some clients may have adifferent leg 2 or 3. Those skilled in the art will recognize either orboth can be served out of cache as required.

FIG. 20 is a flow diagram illustrating actions between the client (userdevice 300), the application 350, the egress router 630, the enforcementnode 150, and the destination 640 in an example operation of theprocesses 650, 660, along with caching of trace results at theenforcement node 150. In this example, the application 350 is the tunnelclient 510 whereas the enforcement node 150 is the tunnel server 520.The flow includes client configuration via the application 350 includingthe cloud tunnel 500. The application 350 can send an ICMP traceroute tothe destination 640 IP address with the Signature-A in the ICMP payload.The enforcement node 150 is configured to terminate the ICMP tracerouteand send an ICMP response by faking the destination IP as the sourcealong with tunnel info in the ICMP payload. Once the application 350 isaware of the tunnel, the application 350 can send a traceroute APIrequest, create an SSL connection with the enforcement node 150 and senda POST request to the tunnel service at the enforcement node 150 withdetails in a JavaScript Object Notation (JSON) body. The application 350can send a restful MTR request to enforcement node 150 which includesthe destination address and port in case of TCP/UDP MTR. It should alsoinclude the MTR type: TCP, UDP or ICMP. The various signatures can bevia a Type-Length-Value (TLV) in the ICMP request and reply.

The enforcement node 150 is configured to perform the reverse trace ofLeg 2 and the trace of Leg 3. The enforcement node 150 maintains theresults of these two Legs 2, 3 in a cache for a predetermined amount oftime, e.g., one minute or some other configurable value. If the resultsare not in the cache, the enforcement node 150 performs the trace, e.g.,using MTR. The enforcement node 150 can combine the results whichinclude latency, packet loss, and hop information and send this via atraceroute POST API response to the application 350.

The application 350 performs an ICMP traceroute to the enforcement node150 outside of the tunnel 500. The application 350 can determine orcompute the Leg 1 results based on subtracting the Leg 2 results fromthe results of this ICMP traceroute to the enforcement node 150 outsideof the tunnel 500. Of course, this can be other types of traceroute.

FIG. 21 is a flowchart of a process 700 for metric computation fortraceroute probes using cached data to prevent a surge on destinationservers. The process 700 is described with reference to one of theenforcement nodes 150 associated with the cloud-based system 100. Theprocess 700 can be implemented as a method that includes steps, via theenforcement node 150 configured to execute the steps, and via anon-transitory computer-readable medium that includes instructions thatcause one or more processors to implement the steps.

The process 700 includes receiving a request, from a client, for one ormore of a first trace of a tunnel and a second trace to a destination(step 701); checking a cache at the node for results from previoustraces of the first trace and the second trace (step 702); responsive tothe results not being in the cache, performing one or more of the firsttrace and the second trace (step 703); and providing the results to theclient so that the client aggregates the results with details from oneor more additional legs to provide an overall view of a service pathbetween the client and the destination (step 704).

The process 700 can further include, subsequent to the performing,storing corresponding results in the cache. The process 700 can furtherinclude, subsequent to a predetermined time period, removing the resultsfrom the cache. The process 700 can further include receiving a tracepacket from the client outside of the tunnel; and providing a responseto the trace packet, wherein the client utilizes details in the responsein addition to the first trace and the second trace to determine detailsof the service path. The process 700 can further include receiving atrace packet to the destination from the client with a signaturetherein; and terminating the trace packet and responding thereto withthe destination's address and with details about the tunnel. The clientcan connect to the destination through at least three legs. Theproviding can include at least one of the first trace and the secondtrace from the cache and the other from the performing.

§ 13.0 TCP Traceroute Using RST and SYN-ACK to Determine DestinationReachability

Referring back to FIG. 11, for a description of a TCP traceroute fromthe client (user device 300) to the destination (node 150), the clientcreates a series of packets with increasing TTL values. TTL values aredecremented for each hop. When each packet is received at the routers602A-602D with the TTL value of 0, the packet is discarded, and aresponse is sent back to the client (“TTL Time Exceeded”). The responseincludes information regarding its location and indicating data transfertimes. Finally, the client knows that the destination has been reached(and stops sending packets) when it receives a different message from ahop, saying that the port intended is unreachable (“Destination/Portunreachable”). In order to use TCP for tracing the path to thedestination, one cannot use standard TCP stream sockets as internallyTCP always retransmits packets, and, as a result, one cannot estimatethe packet loss and latency sitting at the application layer. To avoidthis, traceroute (aka TR) applications use raw sockets where TCP packetsare framed in the application and directly injected into the networkbypassing the TCP stack.

Current TCP traceroute applications/tools cannot determine if thedestination has been reached as they have no ability to read theresponse sent by the destination. In an embodiment, the presentdisclosure includes determining the reachability of the destination bypeeking into the response packets for a SYN-ACK or an RST sent by thedestination. A reception of the SYN-ACK or RST from the destination willindicate the availability of the destination. This ability to peek intothe TCP stack for a response is unique and gives the ability to use TCPas a technique to determine reachability.

ICMP and UDP TR implementations detect the destination reachability bylooking at “ICMP ECHO” response and “UDP port unreachable” errors,respectively. This is relatively straightforward as the responses fromthe intermediate hops and the destination are at the ICMP layer whichthe applications can snoop and process.

TCP poses a unique challenge in that the final destination responds witheither an RST or a SYN-ACK when the TCP SYN hits the destination stack.These responses generated by the destination are not ICMP responses butinstead are standard TCP responses that the local TCP stacks on theoriginator of the request consume. So while the request packet wasinjected by a raw socket, the TCP RST or the SYN-ACK would land up onthe TCP stack and as there is no corresponding TCP socket, the responsefrom the destination is silently dropped believing its a stray. As aresult of this, TCP traceroute applications will not be able to detectthe responses from the destination thus rendering the utility with verylittle use as the path is always incomplete with no destination everdiscovered.

To address the lack of reachability detection of the destination, thepresent disclosure includes a modification to the TCP stack to recognizeTCP traceroute traffic and divert the RST/SYN-ACK response toappropriate “raw sockets” so that the TR application can determine thereachability to the destination. This way the TCP TR can draw thecomplete path with all the intermediate hops and the final destinationgiving the administrator a full picture of the path taken by a packetfrom the source to the destination. Also, the raw RST packet can be sentto the destination as well after SYN-ACK is received by a TR applicationso that the connection can be closed in time rather than waiting for atimeout. As described herein, a TR or traceroute application is softwareexecuted on a processing device such as the server 200 or the userdevice 300 for implementing a traceroute, such as using TCP traceroute.Also, TCP checksum, sequence, and ACK in the RST packet are handled byTR application itself. The source port in the SYN packet is allocated byTCP stack from the port pool based on destination IP and port to avoidcollision with real user traffic.

FIG. 22 is a flowchart of a process 710 for TCP traceroute using RST andSYN-ACK to determine destination reachability. The process 710 isdescribed with reference to one of the user device 300 with theapplication 350 and the enforcement nodes 150 associated with thecloud-based system 100. The process 710 can be implemented as a methodthat includes steps, via a processing device configured to execute thesteps, and via a non-transitory computer-readable medium that includesinstructions that cause one or more processors to implement the steps.The process 710 can be implemented via a traceroute applicationimplementing a TCP stack in the processing device.

The process 710 includes sending a plurality of TCP packets via a rawsocket to perform a trace to a destination (step 711); receivingresponses to the plurality of TCP packets (step 712); detecting theresponses in the TCP stack and diverting the responses to the raw socket(step 713); and aggregating the responses by the traceroute applicationto determine details of a service path from the processing device to thedestination (step 714).

The plurality of TCP packets can include TCP Synchronize (SYN) messages,and the responses include TCP SYN-Acknowledgement (ACK) or Reset (RST)messages. The process 710 receiving a TCP SYN-ACK message from thedestination; and sending a TCP RST packet to the destination. A TCPchecksum, sequence, and ACK in the TCP RST packet can be implemented bythe traceroute application. The raw socket can be used in lieu of a TCPsocket. A port for the raw socket can be allocated by the TCP stack froma pool of ports based on the destination.

§ 14.0 Adaptive Probing to Discover a Protocol for Network Tracing

Traceroute implementations conventionally use just one protocol to tracethe path from the source to the destination along with the hops,latency, and packet loss stats. In an embodiment, the present disclosureincludes a combination of ICMP, UDP and TCP to get a more accuratemeasurements of hops, packet loss, and latency from source todestination. As each network entity tends to respond to a particularprotocol more favorably, the present disclosure uses the protocol thatwould have the highest probability of getting a response. Results fromusing different protocols are aggregated and displayed as one. A problemwith traceroute is that it relies on hosts responding with ICMP errorsfor TTL expiry which is unreliable due to routers either disabling thisor rate limiting. Note, routers that run BGP respond to TCP port 179while blocking ICMP.

The following utilizes the example of FIG. 19 with the three legs,namely Leg 1, Leg 2, and Leg 3. In an embodiment, a singleprotocol—ICMP/UDP/TCP—is used to probe all three legs. Using ICMP/UDPfor Leg 3 is not advisable as the probes are primarily to check theavailability of a destination 640 that is a Web app which is running onTCP ports 80/443. For example, a particular Web app can be 100%available but show a path to the destination that is broken, with thereason being that ICMP and UDP probes are blocked by the destination640.

The present disclosure includes a dynamic probe that tries a combinationof protocol types to get an estimate of packet loss and the latency tothe egress/destination. Determining the intermediate hops and theirlatency/packet loss is a matter of luck irrespective of the protocolused as the TTL expiry is a Layer 3 property handled by routers. Forpractical purposes, the choice of protocol is significant inside acustomer network due to Access Control List (ACL)/Firewall (FW) ruleswhile less significant on the internet although some routers prioritizeTCP traffic over the rest. The choice of protocol is the mostsignificant when the end host receives it as the response to the probeis completely dependent on the rules configured on that host and theseare all over the place.

Most destinations 640 will only respond to TCP ports 80/443. The egressrouters 630 will respond to ICMP-ECHO at times and could either respondwith a SYN-ACK or RST when a TCP probe is sent to port 179/80/443. Thereare only two entities that are guaranteed to respond and metrics tothese can be trusted, and the rest are best effort. The two entitiesinclude the destination 640 responding to a TCP SYN on port 443(assuming Web apps), and the node 150 responding to a PING or TCP SYN.

In an embodiment, the destination 640 is a SaaS endpoint running Webapplications. With a TCP SYN to port 443 on the destination 640, thedestination 640 is bound to respond with a true measure of reachability,latency and packet loss. Assume that this will be the IP of the loadbalancer fronting a server farm for the destination 640 but then that ishow far the service path can be reached. It is also possible to closethe connection to the server with an RST/FIN to free up any resource onthe destination 640. Packet loss and latency to the destination 640 aredetermined by the response to the TCP SYN. One optimization to find thelatency and packet loss could be to harvest the data for the domain fromthe web probes. But it is still necessary to send the TCP tracerouteprobes to determine the number of hops to the destination 640.

FIG. 23 is a network diagram with an excerpt of the network diagram ofFIG. 19 illustrating Legs 2 and 3 for illustrating adaptive probing. Inan embodiment, the egress router 630 is probed from two sides—from theapplication 350 and from the enforcement node 150. The approach is tofirst find a protocol the egress router 630 will respond to by sending aset of probes directly to the egress router 630 by setting a large TTLand then employing the regular MTR logic to trace the hops in between.This way it is known that there is a point at which the probes will geta response.

To give an example, start with ICMP-ECHO to the egress router 630 IPwith TTL=64, if there is no response, then switch to TCP-SYN probes toports 179 (Border Gateway Protocol (BGP)), 80, 443. Either an RST or anSYN-ACK will give the latency and the packet loss.

§ 14.1 Detecting Packet Loss Between the Application and the EgressRouter

There are two parameters to check here—packet loss and latency. In anembodiment, once the egress router 630 IP address is determined,ICMP/UDP probes are sent towards the egress IP with the hope that itresponds. The issue with this is that if the egress router 630 isconfigured to drop ICMP/UDP probes then it will show as unreachable.

With respect to packet loss detection, as the handling of the ICMPresponses to TTL expiry are done in software and rate limited, the lackof an ICMP error response is not a measure of the packet loss at thathop. Also, the egress routers on the customer network might have ICMPturned off or rate limited. But if the packets are being forwarded bythe egress router 630 then that is a good measure of its ability tohandle load and also routers are rated based on their ability to forwardpackets which is mostly done in hardware.

The following describe techniques to gauge packet loss when the egressrouter 630 is configured to drop or rate limit packets.

In a first step, the approach tries to reach the egress router 630 byusing ICMP followed by UDP and TCP and checks for packet loss. This doesnot need to be a configured number of probe, e.g., it can be threeprobes to see if the egress router 630 responds. Based on the responseto a protocol, this is stored for future reference. For example, sendthree ICMP probes and wait for a response. If they all fail, then sendthree UDP problems, and if they all fail, then send three TCP probes.

In a second step, if the result of the first step is not 0% packet lossor an acceptable %, the second step includes trying to reach beyond theegress router 630 to get a response. The intent is to exercise thepacket forwarding path of the egress router 630 versus the softwarehandling of the packets. If the packets could be forwarded successfully,then its implied that there is no loss. A safe reference point can bethe enforcement node 150 as the IP address. There are twopossibilities—approach 1—use the tunnel 500, 610, or approach 2—outsidethe tunnel 500, 610.

In a third step, when the results of the first step and the second stepare not acceptable, pick a last router in the customer's network with aprivate IP that is responding. The egress router 630 is the first publicIP address that is encountered. For the last router, looking at therouting of packets, it is the egress router 630 with one leg in theprivate network and the other in the public that will move the packetout of the customer premise. There could be an independent NetworkAddress Translation (NAT) device before the egress router 630 forNAT'ing the IP but even reaching that could be a fair approximation ofthe loss.

The above steps are performed by the application 350 and it can maintaina cache with the approach and the results that may be refreshedperiodically, when a network change occurs, and/or when the results arenot good. As TCP-SYN seems to be the best bet given the rate limitinglogic for ICMP on most devices, it is possible to a firewall that mightsee too many SYNs going out, and caching seems the best way to avoidraising a False Alarm on the firewalls and for them making changes onthe firewall to let the probes out.

§ 14.2 Detecting Packet Loss Between the Enforcement Node and the EgressRouter

Note that a majority of the IT administrators disable their egressrouters 630 to respond to any form of traffic destined to their IP onthe Internet facing side. Based on experimentation, with ˜7000 egressrouter IP addresses, only 39% responded. In a first approach, the packetloss can be measurement outside of the tunnel 500, 610. Here, theapplication 350 can send a configured number of probes (e.g., ICMP, TCP)to the enforcement node 150, e.g., 11 TCP-SYN probes with TTL=64. Thatis, in this first approach, the assumption is packet loss between theenforcement node 150 and the egress router 630 is the same as the packetloss between the user device 300 and the enforcement node 150. If thepacket loss is zero or acceptable, this is a safe assumption.

In a second approach, the enforcement node 150 can try to direct a traceto the egress router 630. This second approach can be performed if thepacket loss from the first approach is not acceptable. In an embodiment,this can include sending a set number of ICMP probes destined to theegress router IP. If the response is obtained, then ICMP works otherprobes can be sent to the egress router 630 to measure latency andpacket loss. If the ICMP probes fail, then TCP SYN probes can be sent toport 179/80/443 hoping to get a SYN-ACK or RST. Otherwise, UDP probesare sent to the traceroute ports. Any result can be one or a combinationof the first approach and the second approach.

§ 14.3 Detecting Latency from Application and Node to the Egress Router

If the egress router 630 responds, then the latency is known. Theproblem is when the egress router 630 does not and there is still a needto estimate the latency. When switching between the ICMP, the TCP, andthe UDP probes to judge the latency to the egress, if the egress router630 does not respond, the following is performed to infer the latency.

With reference to FIG. 23, it is possible to determine the latency fromthe application 350 to the node 150 as the node 150's IP responds topings and TCP SYN. The latency from the application 350 to the egressrouter 630 is called ‘A’ and the latency from the enforcement node 150to the egress router 630 is called ‘B.’ If either A or B can bemeasured, the other one can be derived and, as long as it is a positivevalue, it can be used as a fair estimate. That is C=A+B, C being thelatency from the client to the enforcement node 150. In the worst case,if the egress router 630 was not reachable from either side, then take‘A’ as the time it takes for the application 350 to reach the farthestrouter (private IP) on the Intranet. If needed, it is possible to takethe time the first public IP took to respond and the time it took toreach the farthest router on the Intranet and average their times.

The reverse trace can be avoided when there is no opaque tunnel present.Here, the application 350 can trace the path from itself to theenforcement node 150 using ICMP or TCP pings. Due to the absence of theopaque tunnel, the traceroute probes from the application 350 will beable to trace its path to the enforcement node 150.

For the purpose of calculating the latency when the application 350 isnot able to reach the egress router 630, it is possible to have theenforcement node 150 to PING/TCP-PING to the egress router 630 to getlatency. The enforcement node 150 does not have to do the traceroute butjust needs to get the Round Trip Time (RTT) to the egress router 630 sothat it is possible to compute A=C−B.

§ 14.4 Comparing ICMP and TCP PING Data

It was evaluated as to whether ICMP and TCP probes take different pathson the Internet. It was determined that TCP and ICMP packets are routedalong the same path on the Internet when we consider the network as anAutonomous System (AS). This was based on a 122 k set of hops and it wasfound that PING and TCP probes took the same path and never deviatedeven once when looking at it from an ASN angle.

§ 14.5 Adaptive Probe Process

FIG. 24 is a flowchart of an adaptive probe process 720 for tracerouteprobes. The process 720 is described with reference to one of the userdevice 300 with the application 350 and the enforcement nodes 150associated with the cloud-based system 100. The process 720 can beimplemented as a method that includes steps, via a processing deviceconfigured to execute the steps, and via a non-transitorycomputer-readable medium that includes instructions that cause one ormore processors to implement the steps.

The process 720 includes, for one or more legs of the plurality of legs,sending a number of probes using one of a plurality of protocols (step721); responsive to receiving a response from the number of probes,determining the one of the plurality of protocols is successful andstoring this protocol the one or more legs (step 722); and, responsiveto failure to receive the response, sending a number of probes usinganother one of the plurality of protocols and continuing until asuccessful protocol is determined or all of the plurality of protocolsfail (step 723).

The plurality of protocols can include Internet Control Message Protocol(ICMP), Transmission Control Protocol (TCP), and User Datagram Protocol(UDP). The plurality of legs can include a first leg, a second leg, anda third leg. The third leg can be to a destination that includes a Webapplication, and wherein a protocol for the third leg includesTransmission Control Protocol (TCP). At least one of the first leg, thesecond leg, and the third leg can include a different protocol usedthereon. Packet loss and/or latency between the first leg and the secondleg can be determined based on a single trace therebetween. The process720 can further include aggregating results for all of the plurality oflegs, wherein at least two of the plurality of legs used a differentprotocol from one another.

§ 15.0 Accurate Differential Traceroute Latency Calculation Between Hops

Again, traceroute is a diagnostic command to find the routes (paths) andmeasures the latency to each hop. In traceroute, each node-to-nodeconnection is called a hop and the latency is the round trip from theuser's machine to the destination.

The conventional traceroute has limitations that it might not becomplete, and the results are not accurate for the final hop as thefinal hop does not provide the processing delay. The traceroute resultsmight not be complete as the final destination might not respond to theprobe. The conventional traceroute does not provide the latency betweenthe hops. Routers typically have a very fast forward path as this isdone in the hardware, but some routers take significant time to respondto TTL expired messages as they do this through software.

In an embodiment, traceroute enhancements are provided that provideaccurate calculations when the traffic goes through the enforcement node150 as well as provides the latency between hops. When a customer usesthe cloud-based system 100, the traffic from the user device 300 is sentthrough the enforcement nodes 150. The traceroute is used to provide thelatency from the user device 300 to the egress router 630 as well to theenforcement node 150. If a site is bypassed in the cloud-based system100, the traceroute measures the latency from the user device 300 to thesite.

The edge connector 150A can be configured to combines this tracerouteinformation with the information from the enforcement node 150 andprovide the measurements to the user. The enforcement node 150 providesthe traceroute measures from enforcement node 150 to the destination640. Both the enforcement nodes 150 and the edge connector 150A cabsupport ICMP, TCP, and UDP protocols for traceroute.

When traffic is going through the enforcement node 150, the edgeconnector 150A can perform the traceroute using the enforcement node150's IP address. The enforcement node 150 is configured to alwaysrespond to the traceroute probe from the edge connector 150A. Thissolves the incompleteness problem for the conventional traceroute thatcan happen in the traceroute that some destinations might not respond tothe probe. If the destination 640 is bypassed in the cloud-based system100, the edge connector 150A does traceroute the destination 640, for abest effort latency measurement to the final destination as the finaldestinations did not provide the processing delay. If the finaldestination did not respond, it provides the information for all otherhops.

When the enforcement node 150 receives this probe, it responds backproviding the packet processing delay in the data payload. This providesaccurate absolute latency to the enforcement node 150. If thedestination is bypassed in the Zscaler cloud, the Zscaler Edge connectordoes the best effort latency measurement to the final destination as thefinal destinations do not provide the processing delay.

§ 15.1 Latency Between Hops

The edge connector 150A sends a configured number of packets to hopsstarting with TTL 1 to the maximum configured TTL to the enforcementnode 150. The hops, which are configured to respond, send the responseand the edge connector 150A measures of the round-trip latency for thepacket to these hops.

The edge connector 150A uses the results from all the routers 602 aswell the enforcement node 150 to calculate the latency differencebetween hops. The edge connector 150A uses the average latency for a hopand uses that to compute adjusted averages and the difference iscomputed between adjusted averages.

§ 15.2 Average Latency

FIG. 25 is a network diagram of a network for illustrating an averagelatency calculation. This section describes how the average latency iscalculated. In this example, there is the user device 300 connected tothe destination 640 via four intermediate routers 602-1 to 602-4. FIG.26 is a diagram of the network of FIG. 25 illustrating an operation.When a router/destination does not respond to ICMP/UDP/TCP tracerouteprobe, the value is recorded as −1. The average (AVG) is the sum of allpositive values divided by the positive value count. If the hop is notresponding, its average latency is set to 0.

The following describes how the average phase is adjusted. The averagelatency for each hop is copied to the adjusted average. The end is thelast hop and the start is the first hop.

Step S1: Set index=end where end is the last value.

Step S2: Set current to end−1.

Step S3: If current==start−1, Go to step 9.

Step S4: If the hop at the current is not responding, setcurrent=current−1. Go to Step S3.

Step S5: If the average latency of the current is more than the adjustedaverage of the index, then set the adjusted average of the current tothe adjusted average of the index. If the average latency for thecurrent is lesser than or equal to the adjusted average of the index,then do not change.

Step S6: Set index=current.

Step S7: Current=current−1.

Step S8: Go to step S3.

Step S9: Exit.

FIGS. 27-30 illustrate an example operation of the average latencyadjustment.

§ 15.3 Differential Average Latency

If there is only one hop, the edge connector 150A can set thedifferential average to its average. The following describes adifferential phase computation.

Step S11: Set index=first responding hop.

Step S12: Set current=index+1.

Step S13: If current=end+1, Go to step S19.

Step S14: If the hop at “current’ is non-responding hop, setcurrent=current+1. Go to step S13.

Step S15: Compute differential average for the hop at current=adjustedaverage of hop at current—adjusted average of the hop at index.

Step S16: index=current.

Step S17: current=current+1.

Step S18: Go to step 13.

Step S19: Exit.

FIGS. 31-34 illustrate an example operation of the differential averagelatency adjustment. This shows that average round trip latency is 14 msfrom the user device 300 to router 602-1. The average latency betweenrouters 602-1, 602-2 is <1 ms. The average latency between 602-2, 602-3is 1 ms. The average latency between the routers 602-3, 602-4 is 2 ms.

§ 15.4 Process for Accurate Differential Traceroute Latency CalculationBetween Hops

FIG. 35 is a flowchart of a process 750 for an accurate differentialtraceroute latency calculation between hops. The process 750 isdescribed with reference to one of the user device 300 with theapplication 350 and the enforcement nodes 150 associated with thecloud-based system 100. The process 750 can be implemented as a methodthat includes steps, via a processing device configured to execute thesteps, and via a non-transitory computer-readable medium that includesinstructions that cause one or more processors to implement the steps.

The process 750 includes performing a plurality of traces between twonodes in a service path (step 751); obtaining latency measurements foreach of the plurality of traces for each of one or more hops between thetwo nodes (step 752); and determining average latency between each ofthe one or more hops based on the latency measurements, adjusted averagelatency for each hop, and differential average latency for each hop(step 753). The nodes can include two nodes in a cloud-based system. Afirst node is an enforcement node 150 and a second node is an edgeconnector 150A. The plurality of traces utilize either Internet ControlMessage Protocol (ICMP), Transmission Control Protocol (TCP), UserDatagram Protocol (UDP), or a combination thereof. A destination of theplurality of traces can be a node in a cloud-based system.

CONCLUSION

It will be appreciated that some embodiments described herein mayinclude one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors; Central Processing Units (CPUs);Digital Signal Processors (DSPs): customized processors such as NetworkProcessors (NPs) or Network Processing Units (NPUs), Graphics ProcessingUnits (GPUs), or the like; Field Programmable Gate Arrays (FPGAs); andthe like along with unique stored program instructions (including bothsoftware and firmware) for control thereof to implement, in conjunctionwith certain non-processor circuits, some, most, or all of the functionsof the methods and/or systems described herein. Alternatively, some orall functions may be implemented by a state machine that has no storedprogram instructions, or in one or more Application-Specific IntegratedCircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic or circuitry. Ofcourse, a combination of the aforementioned approaches may be used. Forsome of the embodiments described herein, a corresponding device inhardware and optionally with software, firmware, and a combinationthereof can be referred to as “circuitry configured or adapted to,”“logic configured or adapted to,” etc. perform a set of operations,steps, methods, processes, algorithms, functions, techniques, etc. ondigital and/or analog signals as described herein for the variousembodiments.

Moreover, some embodiments may include a non-transitorycomputer-readable storage medium having computer-readable code storedthereon for programming a computer, server, appliance, device,processor, circuit, etc. each of which may include a processor toperform functions as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, an optical storage device, a magnetic storage device, aRead-Only Memory (ROM), a Programmable Read-Only Memory (PROM), anErasable Programmable Read-Only Memory (EPROM), an Electrically ErasableProgrammable Read-Only Memory (EEPROM), Flash memory, and the like. Whenstored in the non-transitory computer-readable medium, software caninclude instructions executable by a processor or device (e.g., any typeof programmable circuitry or logic) that, in response to such execution,cause a processor or the device to perform a set of operations, steps,methods, processes, algorithms, functions, techniques, etc. as describedherein for the various embodiments.

The foregoing sections include headers for various embodiments and thoseskilled in the art will appreciate these various embodiments may be usedin combination with one another as well as individually. Although thepresent disclosure has been illustrated and described herein withreference to preferred embodiments and specific examples thereof, itwill be readily apparent to those of ordinary skill in the art thatother embodiments and examples may perform similar functions and/orachieve like results. All such equivalent embodiments and examples arewithin the spirit and scope of the present disclosure, are contemplatedthereby, and are intended to be covered by the following claims.

What is claimed is:
 1. A method implemented by a node associated with atunnel comprising: receiving a request, from a client, for a trace ofthe tunnel; causing the trace inside the tunnel; obtaining results ofthe trace inside the tunnel; and sending the results of the trace insidethe tunnel to the client so that the client aggregates these detailswith details from one or more additional legs to provide an overall viewof a service path between the client and a destination.
 2. The method ofclaim 1, wherein the trace inside the tunnel includes identifying apacket with a predefined signature, analyzing a Time-to-Live (TTL) valuein the packet, and sending a response to a probe initiator based on theTTL value.
 3. The method of claim 2, wherein the response includes anInternet Protocol (IP) address that was spoofed based on a router wherethe TTL value expired.
 4. The method of claim 1, wherein the traceinside the tunnel includes sending trace packets to another end of thetunnel each having increasing Time-to-Live (TTL) values.
 5. The methodof claim 4, wherein the trace packets are sent from both ends of thetunnel to determine any changes in routing between directions.
 6. Themethod of claim 1, wherein the tunnel includes a data channel and acontrol channel each having different encryption.
 7. The method of claim6, wherein the encryption is any of Transport Layer Security (TLS),Secure Sockets Layer (SSL), and Datagram Transport Layer Security(DTLS).
 8. A non-transitory computer-readable medium comprisinginstructions that, when executed, cause one or more processorsassociated with a node that is associated with a tunnel to perform stepsof: receiving a request, from a client, for a trace of the tunnel;causing the trace inside the tunnel; obtaining results of the traceinside the tunnel; and sending the results of the trace inside thetunnel to the client so that the client aggregates these details withdetails from one or more additional legs to provide an overall view of aservice path between the client and a destination.
 9. The non-transitorycomputer-readable medium of claim 8, wherein the trace inside the tunnelincludes identifying a packet with a predefined signature, analyzing aTime-to-Live (TTL) value in the packet, and sending a response to aprobe initiator based on the TTL value.
 10. The non-transitorycomputer-readable medium of claim 9, wherein the response includes anInternet Protocol (IP) address that was spoofed based on a router wherethe TTL value expired.
 11. The non-transitory computer-readable mediumof claim 8, wherein the trace inside the tunnel includes sending tracepackets to another end of the tunnel each having increasing Time-to-Live(TTL) values.
 12. The non-transitory computer-readable medium of claim11, wherein the trace packets are sent from both ends of the tunnel todetermine any changes in routing between directions.
 13. Thenon-transitory computer-readable medium of claim 8, wherein the tunnelincludes a data channel and a control channel each having differentencryption.
 14. The non-transitory computer-readable medium of claim 13,wherein the encryption is any of Transport Layer Security (TLS), SecureSockets Layer (SSL), and Datagram Transport Layer Security (DTLS).
 15. Anode associated with a tunnel comprising: one or more processors andmemory comprising instructions that, when executed, cause the one ormore processors to receive a request, from a client, for a trace of thetunnel; cause the trace inside the tunnel; obtain results of the traceinside the tunnel; and send the results of the trace inside the tunnelto the client so that the client aggregates these details with detailsfrom one or more additional legs to provide an overall view of a servicepath between the client and a destination.
 16. The node of claim 15,wherein the trace inside the tunnel includes identifying a packet with apredefined signature, analyzing a Time-to-Live (TTL) value in thepacket, and sending a response to a probe initiator based on the TTLvalue.
 17. The node of claim 16, wherein the response includes anInternet Protocol (IP) address that was spoofed based on a router wherethe TTL value expired.
 18. The node of claim 15, wherein the traceinside the tunnel includes sending trace packets to another end of thetunnel each having increasing Time-to-Live (TTL) values.
 19. The node ofclaim 18, wherein the trace packets are sent from both ends of thetunnel to determine any changes in routing between directions.
 20. Thenode of claim 15, wherein the tunnel includes a data channel and acontrol channel each having different encryption.